Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training

We address the poor scalability of learning algorithms for orthogonal recurrent neural networks via use stochastic coordinate descent on group, leading to a cost per iteration that increases linearly with number states. This contrasts cubic dependency typical feasible such as Riemannian gradient descent, which prohibits big network architectures. Coordinate rotates successively two columns matrix. When (i.e., indices rotated columns) is selected uniformly at random each iteration, we prove convergence algorithm under standard assumptions loss function, stepsize and minibatch noise. In addition, numerically show has an approximately sparse structure. Leveraging this observation, propose variant our proposed relies Gauss-Southwell selection rule. Experiments benchmark training problem approach very promising step towards